Machine learning classification and training for digital microscopy cytology images

ABSTRACT

The disclosure relates to machine learning classification of cells/particles in microscopy images. A method includes inputting an image having invisible features into an initial neural network classifier (INNC) of a convolutional neural network. The INNC is trained using images with ground truth derived from out-of-channel mechanisms. The method includes generating an intermediate classification from the original image. The intermediate classification and the original image are input into a final neural network classifier (FNNC) that comprises one or more bypass layers to feed forward an initial, final classification from a final activation layer to a final convolutional layer thereby bypassing a final pooling layer. The final convolutional layer has an increased kernel size and more filters than the initial convolutional layer. The final classification is generated based on the invisible features in the original image and outputted.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/500,406, filed May 2, 2017, which is incorporated herein by referencein its entirety, including but not limited to those portions thatspecifically appear hereinafter, the incorporation by reference beingmade with the following exception: In the event that any portion of theabove-referenced provisional application is inconsistent with thisapplication, this application supersedes said above-referencedprovisional application.

The present disclosure relates to systems and methods for identifyingparticles and more particularly relates to systems, methods, andplatforms for imaging and machine learning classification or detectionof cells and particles in cytology images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic diagram of a system for imaging andmachine learning classification or detection of particulates ormaterials in accordance with the teachings and principles of thedisclosure;

FIG. 2 illustrates a schematic diagram of a sample slide in accordancewith the teachings and principles of the disclosure;

FIG. 3 illustrates is a schematic diagram illustrating a section andsubsections of a digitized image obtained of a sample in accordance withthe teachings and principles of the disclosure;

FIG. 4 is a schematic block diagram illustrating operation of aclassification system in accordance with the teachings and principles ofthe disclosure;

FIG. 5 graphically illustrates a method for classifying or detectingparticles in an image to generate a heat map or bounding boxes inaccordance with the teachings and principles of the disclosure;

FIG. 6 is a schematic flow chart diagram illustrating a method forclassifying or detecting particles or materials in a sample inaccordance with the teachings and principles of the disclosure;

FIG. 7 is a schematic block diagram illustrating an embodiment of aneural network for classifying particles or particle types in accordancewith the teachings and principles of the disclosure;

FIG. 8 is a schematic block diagram illustrating another embodiment of aneural network for classifying particles or particle types in accordancewith the teachings and principles of the disclosure;

FIG. 9 is a schematic block diagram illustrating classifying particlesor particle types in accordance with the teachings and principles of thedisclosure; and

FIG. 10 illustrates a block diagram of an example computing device inaccordance with the teachings and principles of the disclosure.

DETAILED DESCRIPTION

The present application discloses systems, methods, and devices forobtaining and classifying or detecting particulates or materials usingdigital imaging, such as digital microscopy. Information about theparticles or materials present in a location or environment may be ofgreat interest for research, medical, or other purposes. For example,particles in a body, soil, living environment, or any other location maybe used to determine information about medical or ecology health,forensics, or the like. Example sample types include tissue, bone,fecal, blood, urine, sputum, infection disease, cervical mucus, vaginalfluid, milk, semen, geological, forensics, agricultural samples, orinsect or small animal samples. One use for microscopy images is incytology or cytopathology where cells are examined for identification ordiagnosis of a condition in a patient.

According to one embodiment, samples of particles or materials may beobtained and placed on a sample slide. Sample slides may include slideson which a sample material can be placed for review and imaging. Forexample, a sample may include particulates that were suspended orlocated in air, liquid, soil, body or plant tissue, or on a surface in abuilding, furniture, appliance or other location. The sample slide mayinclude a transparent slide that may be used to protect and/or view theparticulates captured in the sample. For example, a technician or usermay obtain a sample and place the sample on a slide.

Information in addition to the slide may also be obtained. For example,a user may also provide other information about the location orenvironment where the particles or material were obtained. Additionally,other information may be included such as health symptoms, ages, aserial number linked to a location or customer, a type of sample, or thelike.

The slides and other information may be received from a user, customer,technician, or other entity that has obtained and forwarded one or moresamples. For example, a lab worker may receive the one or more slidesand load the slides into a scanner for imaging. In one embodiment, thelab worker may scan a barcode on the slide that links the slide withother information about the slide (e.g., a customer, location, healthsymptoms, and sample type). The barcode may be used to automate where tolook on the slide to locate the particulates from the sample. Forexample, the barcode may identify a manufacturer, customer, or party orentity that obtained the slide because the manufacturer, customer, orparty or entity that obtained the slide may indicate where the sample isactually located. In some cases, it can be difficult to locateparticulates on a slide, such as mold spores, if you don't know wherethe sample was placed on the slide. For example, the slide may be muchlarger than the actual sample so it is often efficient to onlyscan/image the portion of the slide where the particulates are located.Knowing the entity, customer, or slide manufacturer (or brand) may allowa scanning system to automate location and scanning of the relevantportion of the slide.

Samples may be imaged using a wide range of different imaging techniquesand at a wide range of different magnifications. Example scanners orimagers that may be used include a digital microscope, bright-fieldmicroscope, polarized imager, phase contrast image, fluorescence imager,scanning electron microscope, dark-field microscope, or other types ofscanners/imagers. During scanning or imaging of the sample, the scanner(such as a digital microscope) may be used to scan or image the wholearea where the sample is located (e.g., where any particulates ormaterials are located). These obtained images may be quite large inpixel count and memory size. For example, the images may be infull-color (16 bit, 24 bit, 32 bit or more) with very high resolution(pixel count and/or dots per inch). In one embodiment, theimaging/scanning process obtains not only images of the whole area, butalso images at different resolutions. For example, the sample area maybe divided up into a grid of smaller sized areas, which are each imagedat a high magnification and then multiple grid areas may be imagedtogether at a lower/wider magnification. Different magnifications may behelpful in imaging/identifying different sizes of particulates ordetecting details for identifying different material types. For example,a single sample may include particles of different sizes that would behelpful to detect and identify.

After imaging, the resulting digital images may be stored or associatedwith a serial number identifying the location where the sample wastaken, a customer that obtained the image samples, or any otherinformation about the location, sample, type, study type, medicalconditions, or the like.

The digital images may be stored and transmitted to a cloud storage orremote storage location for training, aggregation, analysis,classification, association or aggregation with other data, or the like.For example, the lab may acquire the images and upload to a file server.The file server may include a listener that detects the uploading of newimages and uploads those to a remote classification system forclassification, storage, reporting, and/or sharing. Related data mayalso be uploaded with the image for storage at a remote or cloudlocation. Custody tracking of the sample, digitized images, andassociated data may be provided to ensure security and accuracy of thedata. In one embodiment, images, customer information, healthinformation, or the like may be associated with a common serial numberor other identifier so that correlations between various data can bedetermined.

Data stored in the remote or cloud storage may include data, includingimages and related data, from a large number of different labs,customers, locations, or the like. The stored data may be accessible toa classification system that includes a classification model, neuralnetwork, or other machine learning algorithm. The classification systemmay classify each image (or sample associated with the image) asincluding a particular type of particle. For example, the classificationsystem may analyze each image to classify or detect particles within theimages. A particle may be classified as a particular type of particle,such as a particular type of tissue, crystal, cell, parasite, bacteria,mineral, pollen, or the like. For example, the classification system maygenerate a heat map for an image indicating which regions of the imageinclude different types of particles.

As another example, the classification system may generate boundingboxes and then detect/classify particles in the bounding boxes. Forexample, the bounding boxes may indicate regions where there is likelysomething to be present (e.g., a mold spore) to be classified ordetected. This may allow analysis or processing of only portions of theimage using a neural network or algorithm to locations where particlesor materials are present and ignoring regions where there are notparticles. For example, large regions of an image may be blank or whitespace where no particles are present. Processing blank or white spaceregions using neural networks may be a waste of computing resources. Themodel or neural network used by the classification system may include asystem trained based on human classified images or samples. For example,the neural network may be trained using supervised learning. In oneembodiment, learning and training may be performed using unsupervisedmachine learning.

In one embodiment, the classification system may provide one or moreimages (or portions of images) of samples to users for classification.For example, previously classified or unclassified images may beprovided to one or more experts for classification. The experts mayprovide their own classification, which may be used to either confirm orchange an original classification. The machine learning algorithms,models, or neural networks may be retrained based on the updatedclassifications to further improve machine learning models andalgorithms. In one embodiment, changes to classifications for specificparticles or images may be tracked. Tracked classifications may provideadditional data about the accuracy of classifications and can lead tofurther refinement in machine learning and classification algorithms andmodels.

Based on classification of particles within samples, reports may begenerated for the type of study that is being performed for a location,patient, or customer. The report may be generated based on theclassification of particles within images of samples, particle countsfor different particle types, health conditions related to the presenceor counts for specific particle types, or the like. The report may beautomatically generated specific to the serial number, customer, and/orlocation associated with the images and the corresponding sample. In oneembodiment, a report may include a report for the types of particlesdetected, the number of particles, likely conditions in the sampledenvironment or patient, recommended steps to be performed, or the like.The report may be provided as a general report for a specific particletype or may be more general to health or conditions for a sampleenvironment.

As used herein, the term “particle” is given to mean any small unit orportion of material such as dust, mold sports, cells or groups of cells,fibers, small chunks of materials, organism(s), tissue, biologicalmatter, minerals, or any other item or material discussed herein asbeing classified or detected. Additionally, the classification,detection, or identification of particles may include identifying aspecific type of particle or condition of a specific particle ormaterial. For example, cells may not only be identified as a specificcell type, but also as having or displaying a certain condition, such asa condition that corresponds to an abnormality or to cancer.

Embodiments disclosed herein may provide significant utility andbenefits. For example, automated particle classification, reportgeneration, and/or the like may significantly reduce expert time and/orerrors (such as typographical errors), thereby increasing efficiency andaccuracy. At least some embodiments disclosed herein enable the fullclassification of each particle within a whole sample. Generally,technicians do not have enough time, nor are they required to, analyzeevery particle or the full sample for particle type or classification.Additionally, it can take a large amount of time for a technician toperform particle analysis and classification within a sample. This timecan be saved by using machine learning algorithms and/or deep neuralnetworks for automated computer or machine learning classification.Accuracy may be increased because a greater portion of the slide (orsample) is actually analyzed and because embodiments of machine learningalgorithms or models may provide greater classification accuracy for aparticle type and even for a larger number of different particle types.

Embodiments disclosed herein further allow for the long-term storage anduse of samples because they are stored as digitized images and stored ina central location. Machine learning algorithms may be refined based onthe large corpus of data and thus improved particle identificationalgorithms and machine learning results may be obtained. Error in reportcreation may also be decreased because typographical errors by humansmay be reduced or eliminated. For example, even if a generated report ismodified by a human user or technician after generation, the reportdetails may reduce the chance of filling out a report with informationfrom an incorrect sample, customer, location, or the like. Furthermore,the tracking of the movement of the sample, digital images, associatedhealth or location data, changes in particle classification, or the likemay lead to quicker and more accurate reporting to clients.Additionally, more accurate tracking may lead to less managementoverhead and reduce the amount of time it takes to place a process in afinal state so that a report and/or bill may be sent to a customer.Thus, a larger number of customers may be served and even betteridentification and reporting results may be obtained.

In some cases, distinctions between different particles, or distinctionsbetween different classifications of the same particle type may not bedetectable visually for humans. Because existing classification methodsfor particles depend on human classification using the same channel asthe human (e.g., visual detection/classification based on an image),existing systems and methods are unable to distinguish between particlesthat are indistinguishable to humans. However, Applicants haverecognized that, at least in some cases, machine learning algorithms andmodels, such as deep neural networks, can be trained to perform betterthan humans in visual classification and detection. As such, Applicantshave developed, and herein disclose, systems, methods, and devices forparticle classification using out-of-channel means for establishingground truth. The non-image data may include any out-of-channel meansfor determining the classification for a particle or cell. Exampleout-of-channel means for classifying a cell or particle in cytology (inat least one embodiment) include: molecular tests such as polymerasechain reaction (PCR) tests, next generation sequencing, or any othermolecular test; non-bright field microscopy images such as fluorescence,fluorescence in situ hybridization (FISH), electron microscopy, phasecontrast, polarized light, dark field, or any other type of images;genetics and cytogenetics testing; flow cytometry; cytochemistry;antibody testing such as western blot, enzyme-linked immunosorbent assay(ELISA), or other antibody testing. Other out-of-channel means may alsobe included and may include any non-image means of determining aclassification, in some embodiments. Just because a human is not able toidentify a visual distinction based on an image does not mean that theimage does not include features or data that can be used by a neuralnetwork other machine learning algorithm to distinguish betweenclassifications. Because out-of-channel means may be used to reliablyclassify a particle, the out-of-channel means may be used as groundtruth for training. For example, an out-of-channel means may be used totell something about the cell that isn't detectable by a human (trainedor not) using bright-field microscopy.

With ground truth for a specific cytology specimen or sample derivedfrom an out-of-channel means, an image of the specimen or sample (suchas a bright-field microscopy image) may be provided with the groundtruth during a training procedure or process. In one embodiment, aconvolutional neural network may be trained using a set of this data todetect features or details in the bright-field image that are difficultor impossible for even trained humans to detect. In one embodiment, amethod for training a machine learning model includes obtaining andimaging (scanning) a slide with a sample that is known to have aparticular disease by using an out-of-channel mechanism. The methodincludes locating and/or cropping cells from within the image. Themethod includes classifying the cells into various classifications(e.g., cell types). One or more of the classifications of cells (e.g.,all leukocytes) are included in a training set. The imaging,locating/cropping, and classifying processes are repeated for a healthyperson and/or for one or more additional diseases. A resulting trainingset may include the same type of cell with a corresponding known disease(known based on the out-of-channel mechanism). The method includes usingmachine learning and this training data to create a model or algorithmthat can predict the disease of a host based off of one or more cells ofthe specific cell type (e.g., leukocytes). Example machine learningmodels and algorithms include a convolutional neural network (CNN), deeplearning algorithms and networks, or computer vision. In one embodiment,the model or algorithm may be designed according to embodimentsdiscussed herein.

In one embodiment, improved detection of diseases and pathologies usinga trained model or algorithm may be achieved using easily obtainablecells. For example, Applicants have recognized that leukocytes within ablood draw may be used to detect diseases and conditions that werepreviously only detectible using more expensive sampling and testingmethods. Improved detection of cytology and cytopathology usingmicroscopy images and machine learning significantly improves speed,reduces cost, and improves accessibility for disease diagnoses.

A detailed description of systems and methods consistent withembodiments of the present disclosure is provided below. While severalembodiments are described, it should be understood that this disclosureis not limited to any one embodiment, but instead encompasses numerousalternatives, modifications, and equivalents. In addition, whilenumerous specific details are set forth in the following description inorder to provide a thorough understanding of the embodiments disclosedherein, some embodiments may be practiced without some or all of thesedetails. Moreover, for the purpose of clarity, certain technicalmaterial that is known in the related art has not been described indetail in order to avoid unnecessarily obscuring the disclosure.

FIG. 1 is a schematic block diagram illustrating a system andenvironment for obtaining, classifying or detecting, and/or sharingdigitized images of particle or material samples, according to oneembodiment. The system 100 includes an imaging system 102 and aparticulate classification system 104. In one embodiment, samples may beobtained from a location or environment by a technician or professional106.

The technician or professional 106 may include a person that has beentrained or licensed to obtain samples. The samples may include samplesfrom any source, location, or environment disclosed herein. For example,an air sample, tape sample, surface sample, or other sample that maycapture mold spores may be obtained. As further examples, samplesrelated to health or medical services may be obtained from a patient oranimal by a medical professional or veterinarian. Samples from any ofthe locations, environments, or materials discussed herein may beobtained by other professionals, lay persons, or customers in any mannerdiscussed herein or known in respective industries or fields. Examplemethods to obtain samples include obtaining a portion of whatevermaterial is to be imaged or processed such as bodily fluids or tissues,plant matter, soil, lake or other water sources, dust, surface samples(e.g., tape samples or swabs). In one embodiment, the sample may bestored in a container such as a vial, bag, bottle, or other container.

A technician or professional 106 may also obtain any other informationabout the sample including a source of the sample, a location, a sampletype, a study or testing purpose, a customer or patient identifier, adate, temperature or weather conditions, or any other information aboutor related to the sample. This information may be logged by thetechnician or professional 106 into a computer system or other databaseand associated with the sample, such as by associating the sample,customer, or patient identifier. This information may be provided toand/or stored in the classification system 104 and/or a data store 112.The technician or professional 106 may obtain the sample in response toa request or approval by a customer or patient. The customer or patientmay agree that the data will be used for research/big data purposes.

After the samples are obtained, the samples may be applied to a slide108 for preservation, shipping, standardization, or the like. The slide108 may be provided to a lab or an imaging system 102 for imaging ordigitization of the sample. When the sample is received by the lab ororganization that is to digitize the sample, the lab logs the samplesinto the lab's software to a customer's job. The customer at this pointmay be the original customer, patient, or the inspecting organization. Acustomer account may be created in the labs software and/or in softwarefor the classification system 104. The lab may create a job for thesample or customer in the lab's or particulate classification system'ssoftware. The software may link the sample to the customer and job basedon a sample serial number.

If a lab or customer does not have an account with the classificationsystem 104, an account may be created and any necessary software may beinstalled on a lab computer. For example, a user account withinformation about the lab or customer may be generated andauthentication details may be provided for using the account. As anotherexample, the classification system 104 or another party may, at times,need to uniquely identify the scanner or lab from which the digitalsamples will be provided.

When the slides are received and a job is created in a correspondingaccount, the lab may prepare the slide for digitization. For example,the slide may be stained, covered with a cover slip, or the like, asnecessary. The slides may be loaded into the scanner (such as a digitalmicroscope or other imaging device discussed herein). Depending on thetype of sample or manufacture for the sample media, the slide may beloaded such that a pre-selected area will be scanned. For example,samples or slides for different sources (such as customers ormanufacturers) may position the sample consistently at a specificlocation in the slide. This can make it easier for a technician or theimaging system 102 to determine what portion of the slide to image. Forexample, the particulates or materials of a sample can be very small andmay not be visible to the naked eye. Thus, it can be difficult for auser to see what portion of the slide needs to be imaged. If the sampleis a tape sample or other sample that may not have a consistentlocation, a user may need to select the sample size and set the focuspoints manually. The scanner then images the sample and automaticallyloads the sample images up to the classification system 104 over anetwork 110. The sample images may be stored within or associated to theaccount of a lab or customer account and/or may be associated with anyother data gathered about or in relation to the sample.

The classification system 104 may receive images from the imaging system102 and/or images from a large number of imaging systems 102 for thesame or different labs. In one embodiment, the images may be stored in adata store 112. In one embodiment, once the images are processed and/orstored, a notification may indicate that there are new samples ready tobe processed.

In one embodiment, samples, or particles within samples, may beautomatically classified by a deep neural network or other machinelearning classification algorithm of the classification system 104. Forexample, images may be fed to a neural network classifier (e.g.,structured and/or trained in a manner disclosed herein), which outputsan indication of what type of particle(s) is/are in the image. In oneembodiment, a scaled or raw subsection of an image having pixeldimensions required by the neural network or classification algorithmmay be provided to the neural network or classification algorithm. Inone embodiment, the same image or section of image may be processed by aplurality of neural networks to determine what types of particles are inthe image. The classification algorithm or network may indicate aprobability that the image includes particles of a particular type.

The classification system 104 may process all images correlating to asample imaged by the imaging system 102. For example, the classificationsystem 104 may use a machine learning algorithm or model on all imagesand/or all portions of the image to obtain an extremely accurateclassification of most or all particles within the images. This providesconsiderable accuracy and completeness because human technicians, evenif they can determine a classification or diagnoses based on an image,generally don't have the time or ability to analyze all portions of asample where particulates are located. With machine learning processingof each image corresponding to a sample, an extremely accurate particlecount, particle type count, and the like may be achieved.

In one embodiment, samples, or particles within samples, may beclassified or reviewed for classification by a human worker. Forexample, as a neural network or machine learning algorithm is beingtrained, human review or initial classification of a particle may berequired. When samples are ready to review, samples may be grabbed andworked on by a technician working on a local or remote computer 114.After selecting the sample from a list needing attention, the technicianis presented with the cut out particulate images. For example, theimages may be cropped in order to provide an image where the particle ormaterial is large enough to be viewed or observed by the user. Thetechnician may be able to provide an initial classification, or reviewand re-classify or modify a classification given by a neural network,which may then be fed back into the machine learning as new samples forfurther training of a model or network. For example, an image may bereclassified based on an out-of-channel mechanism.

In one embodiment, images may be shared by a technician with a colleagueby clicking on a share option. The image may be sent by theclassification system 104 via email to a registered user with a linkthat takes them to the image. During the classification stage, any notesfrom the technician could be added to an image or section of the image.If the context of the image within the sample is needed for the report,the user may indicate the larger area surrounding the sub section to besaved to notes to be included as part of the report or for later use.The technician may provide an indication that gives their approval thatthe sample has been reviewed and that a particle or material iscorrectly classified. In one embodiment, a technician could repeat thisprocess until all of the slides/images pertaining to this particularcustomer job have been reviewed and approved.

The classification system 104 may include a reporting component forgenerating reports. The classification system 104 may automaticallygenerate a report based on the identification of particles in an imageof the slide and/or a report, investigation, or examination typecorresponding to the slide or sample. The report may include informationabout particle types, particle counts, health impacts, instructions tocheck for certain health symptoms, potential remedies to reduce orincrease particulate counts, or the like. In one embodiment, a reportgeneration process can be started by selecting a button or menu on alist of jobs (e.g., customer jobs for sample processing). Theclassification system 104 may generate a report template that shows thetypes of materials and particulates found, a customer or serial number,account information, health symptoms, or any other information relatedto a specific report or examination type. For example, the templatereport may include health concerns if these particulates are found andpotential investigation/remediation steps.

In one embodiment, the generated report or report template may beprovided to a technician for review, confirmation, and/or transmissionto a client. The technician or other report writer may be able toprovide input to modify the report, add pictures to the report from alist of saved pictures for that customer, save and sign a report, and/ortrigger sending of the report (such as via email in a .pdf file) acompany or customer that requested the report. Once the report is sent,lab management software of the imaging system 102 or classificationsystem 104 is notified that the report has been delivered.

In one embodiment, the classification system 104 may export informationfor external report generation. For example, the classification system104 may be able export information in a format or condition that anothersoftware program can generate a report based on the information. In oneembodiment, if a user prefers not to use the built-in report writer, theuser should be able to export the data and pictures to lab managementsoftware of the user's choice. The text information may be exported in a.csv file or .xml file, in one embodiment. In one embodiment, the imagesmay be exported to a unique directory or file for access by the externalreport generation program or user.

Upon classification and/or reporting, a specific job may be marked ascomplete. In one embodiment, the information may be archived. Forexample, 6 months after a completion date for the job, the customer jobmay go to an archived state for 5, 10, 20, or more years. In oneembodiment, the data may be kept indefinitely for research use or astest/research data for improving a machine learning or deep learningalgorithm or model. In one embodiment, archived information may besearched based on any relevant information, such as location, barcode ofslide, or any other information associated with an image, slide,customer, or job.

In embodiments where the classification system 104 is accessible from aremote location, such as via the Internet, significantly improvedmachine learning and classification may be possible. For example, inmachine learning applications the cost of obtaining data and obtainingannotations of the data (e.g., an indication of a classification) can beextremely time consuming and/or difficult to obtain. The remotelyaccessible classification system 104 may train algorithms based on allimages of the same type of material and thus those accessing theclassification system 104 may obtain the benefits of large datasets thata party may not otherwise be able to obtain. For example, some types ofexaminations may not occur frequently enough within a given organizationto obtain enough data to train a machine learning model or neuralnetwork. By sharing data among different locations and evenorganizations, even examinations that occur infrequently for eachorganization may occur frequently enough in combination to adequatelytrain machine learning models or networks.

FIG. 2 is a schematic diagram illustrating an example sample slide 200,according to one embodiment. The slide 200 includes a region 202 wherethe sample is located. For example, a tape sample, air sample, liquidsample, smear, or other particles from a sample may be located only inthe region 202 so that it is not necessary to image any other portionsof the slide 200. The slide 200 also includes a label 204 which mayinclude a barcode, serial number, brand, or other identifier. Theidentifier may be used to determine that the sample is located withinthe region 202. For example, the imaging system 102 or technician mayknow or determine where to position and/or scan the slide 200 based onthe label 204.

FIG. 3 is a schematic diagram illustrating a section 300 and subsections302, 304 of a digitized image obtained of a sample slide, according toone embodiment. The section 300 may include a portion of the region 202of the slide 200, subsection 302 includes a quarter of the section 300and subsection 304 includes a sixteenth of section 300. In oneembodiment, full resolution images of the section 300 may be obtained atdifferent magnifications. A first magnification image may be obtained ata wider zoom to capture the whole section 300. Four images at a secondmagnification covering a region the size of subsection 302 may beobtained of the section 300. Sixteen images at a third magnificationcovering a region the size of subsection 304 may be obtained of thesection 300. Thus, a plurality of images of the same region, atdifferent magnifications may be obtained. The different magnificationsmay be useful in identifying or classifying or detecting different sizeparticles. For example, mold spores are generally smaller than insectsand thus may require a higher magnification image to accurately detector classify. Other magnifications may be used to obtain desired levelsof detail based on the specific types of particles or materials that areto be detected. For example, cytology images that are expected to beused to detect or classify a specific type of cell may use a zoom factorthat allows that specific type of cell to be adequately images.

FIG. 4 is a schematic block diagram illustrating operation of aclassification system 104, according to one embodiment. In oneembodiment, a network or machine learning algorithm 402 (which may alsobe referred to as a hypothesis), may be trained and used for identifyingand classifying or detecting particles in an image. The network ormachine learning algorithm 402 may include a neural network, such as adeep convolution neural network, or other machine learning model oralgorithm for classifying or identifying particle types.

In one embodiment, the network or machine learning algorithm 402 istrained using a training algorithm 404 based on training data 406. Thetraining data 406 may include images of particles or materials and theirdesignated classifications. For example, the training data may includeimages classified as including particles or materials of a first typeand images classified as including particles or materials of a secondtype. The images may include cropped images that contain a specificspecimen, cell, or particle that has the indicated classification. Inone embodiment, the classification may be determined based onout-of-channel mechanisms, such as those discussed herein.

The types of the particles or materials may vary significantly based onthe type of examination or report that is needed. Training data for anyother type of particle, material type, or the like may be used. Forexample, training data for any particles that are to be identified bythe machine learning algorithm 402 may be provided. Using the trainingdata, the training algorithm 404 may train the machine learningalgorithm 402. For example, the training algorithm 404 may use any typeor combination of supervised or unsupervised machine learningalgorithms.

Once the network or machine learning algorithm 402 is trained, thenetwork or machine learning algorithm 402 may be used to identify orpredict the type of particle within an image. For example, anunclassified image 410 (or previously classified image with theclassification information removed) is provided to the network ormachine learning algorithm 402 and the network or machine learningalgorithm 402 outputs a classification 412. The classification 412 mayindicate a yes or no for the presence of a specific type of particle.For example, the network or machine learning algorithm 402 may betargeted to detecting whether a specific type of mold, bacteria,particle, cell feature, cell exhibiting a specific type of pathology, ormaterial is present in the un-classified image 410. Alternatively, theclassification 412 may indicate one of many types that may be detectedby the network or machine learning algorithm 402. For example, thenetwork or machine learning algorithm 402 may provide a classificationthat indicates which type of particle is present in the un-classifiedimage 410. During training, the classification 412 may be compared to ahuman classification or an out-of-channel classification to determinehow accurate the network or machine learning algorithm 402 is. If theclassification 412 is incorrect, the un-classified image 410 may beassigned a classification from a human and used as training data 406 tofurther improve the network or machine learning algorithm 402.

In one embodiment, both offline and online training of the network ormachine learning algorithm 402 may be performed. For example, after aninitial number of rounds of training, an initial accuracy level may beachieved. The network or machine learning algorithm 402 may then be usedto assist in classification with close review by human workers or inconjunction with separate out-of-channel testing. As additional datacomes in the data may be classified by the network or machine learningalgorithm 402, reviewed by a human, and then added to a body of trainingdata for use in further refining training of the network or machinelearning algorithm 402. Thus, the more the network or machine learningalgorithm 402 is used, the better accuracy it may achieve. As theaccuracy is improved, less and less oversight of human workers may beneeded.

FIG. 5 provides a graphical representation of classifying or detectingparticles in an image 502 using a sliding window 504 to classifysub-portions of the image and generate a heat map 506. In oneembodiment, the classification system 104 may generate the heat map 506by analyzing a portion of the image 502 within a current position of thewindow 504 using the network or machine learning algorithm 402 toclassify or detect particles within the window 504. For example, thewindow 504 may start at the upper left corner of the image 502 and thenetwork or machine learning algorithm 402 may output a classificationfor that region. The classification may include an indication of theparticle type, an indication that there is a particle, and/or anindication that there is no particle at the position (e.g., blank whitespace).

After classification at that position, the window 504 may be moved overto the right. The new position may immediately border or partiallyoverlap with the previous position of the window 504. The section of theimage 502 within the window 504 at the new location may be analyzed. Theclassifications may be tracked for each position as the process isiterated to move the window 504 across and down (or in any othersystematic or random pattern) so that every section of the image 502 hasbeen within the window 504 at least once during the process. Based onthe classification for the window at each position, a heat map 506 maybe output. The heat map 506 includes a plurality of regions 508(designated by regions surrounded by dotted lines) that indicate regionswhere certain types of particles have been detected. In one embodiment,each region 508 may have a different classification, as determined bythe network or machine learning algorithm 402. For example, the heat map506 may indicate that a first type of particle is located at a firstlocation and that a second type of particle is located at a secondlocation. In one embodiment, the section of the image 502 may beanalyzed using an additional sliding window of a different size. Forexample, larger sliding windows may be used to detect particles and/orparticle types of different sizes.

Based on the heat map 506, the sample may be classified has having thetypes of particles identified during the classification process. In oneembodiment, cropped images including the regions 508 may be generatedand stored as examples of specific types of particles. In oneembodiment, the cropped images may be reviewed by a human reviewer orexpert for quality control or may be used on a report to illustrate theparticles that are present in an image or sample.

In one embodiment, instead of generating a heat map, the classificationsystem 104 may generate bounding boxes and then executeclassification/detection algorithms on the content of the boundingboxes. For example, the classification system 104 may use a slidingwindow to detect regions where particles are present to generate aplurality of bounding boxes. Since some of the bounding boxes may beduplicates or overlap, the classification system 104 may remove orcombine duplicate/overlapping boundary boxes to form a subset or new setof bounding boxes. Those bounding boxes (or the pixels in those boundingboxes) may be fed separately into neural networks for processing and foridentification/detection of particles. As one example, bounding boxesmay be placed around all cells or particles of a specific type. Imagedata within the bounding boxes (e.g., cropped images containing the cellor particle) may then be fed to a neural network for detecting aspecific type of classification, such as detecting whether a cellexhibits a specific pathology or characteristics corresponding to aspecific disease.

Similarly, the methods and steps illustrated in relation to FIG. 5 mayalso be used on training data to extract particles for training ordetermining ground truth during a training step.

Referring now to FIG. 6, a schematic flow chart diagram of a method 600for classifying a sample is illustrated. The method 600 may be performedby a classification system, such as the classification system 104 ofFIG. 1.

The method 600 begins and a lab receives 602 a sample. An imaging system102 digitizes 604 the sample into one or more images at one or moremagnifications. A classification system processes 606 the digital imageusing a machine learning prediction model to classify or detect one ormore particulates or materials of the sample. The classification systemstores 608 the digital image with an indication of a classification forthe one or more particulates or materials. A report component generates610 a report for the sample based on the classification for the one ormore particulates or materials.

In some cases, as mentioned previously, machine learning models andalgorithms may be able to detect visual features that correspond todiseased or abnormal cells even if those visual features are notdetectable by a human. For example, Applicants have found thatneutrophils look different in healthy versus diseased patients to aproperly trained neural network even when a trained human cannotconfidently detect the difference. This allows detection of a diseasedstate (either as a conclusive diagnosis or as a screening) using drawnblood instead of more expensive testing. This can lead to significantcost and time savings because blood draws are much easier, cheaper, andless invasive than many other procedures (such as those that depend onobtaining bone marrow).

In one embodiment, these types of differences with blood cells (or othercells in blood), or other cells may be detected by a trained machinelearning model or algorithm. For example, blood may be drawn from apatient who has tested positive to a specific type of disease using anytype of out-of-channel testing discussed or contemplated herein. Then,with labels or ground truth derived from the testing, images of thedrawn blood (or other cells obtained from a human) may be providedduring a training procedure to allow a machine learning model oralgorithm to identify differences that are not known by a human user oroperator (or even a trained human).

In one embodiment, a conventional deep neural network (DNN) and/orconvolutional neural network (CNN) may be used to achieve sufficientaccuracy in classifying cells as coming from a diseased or anon-diseased host. In one embodiment, specialized structures andconnections between layers may be used to improve detection of featuresin images that would otherwise not be detected by a human physician ortechnician.

CNNs and its variants that include localization, segmentation, andfeatures maps are typically used to recognize, segment, and/or locateimages that humans can identify. A CNN a general class of machinelearning algorithm (or model) modeled roughly on a biological neuralnetwork for visual perception. There are many variations of CNNs, butthey generally share the following characteristics: layers ofconvolution nodes, receptive fields, shared weights, and pooling layers.Convolution nodes are the computational unit in a CNN. A convolutionnode is a node that computes the convolution operation on its inputsusing learned filter weights. Convolution is a powerful mechanism fordetecting visual features in its input field. Convolution nodes aregrouped into a layer, such that all the nodes in a layer operateindependently but share the same input field. Various combinations ofconvolutional and full-connected layers are used with a pointwisenonlinearity applied at the output of each node in that layer.

Convolution nodes process only a small input patch (call its receptivefield) and thus are only required to be sensitive to a very local regionof its input space, which means it can focus all of its learningresources to processing local features in its (relatively small)receptive field. With CNNs the filter for each convolution node isreplicated across the entire field with the replicated units sharing thesame filter weights, which allows for far fewer weights in the modelwhile providing for translation invariance. Translation invariance isthe ability to detect features regardless of their position in the inputfield. CNNs frequently make use of a pooling operation, typicallyimplemented as a layer in the network, which is a form of nonlineardown-sampling. Each pooling layer serves to progressively reduce thespatial size of the network's representation, which has the effect ofenforcing the notion that the absolute location of a feature is lessimportant than its location relative to other features.

CNNs may be used in machine learning systems for the following types ofvisual tasks: detection, classification, segmentation, localization, andregression. In each of these tasks the network makes use of the generalcharacteristics described above, with minor variations specific to thetask type. A learning rule is used to process a sample of input examples(the training set) with human-supplied ground-truth information for eachinput example as the network's target output for that input. Thelearning rule is a kind of numerical optimization algorithm thatiteratively modifies the network connection weights to minimize thedifference between the network's output and the ground-truth output foreach input example in the training set.

A high-level explanation for how a CNN performs a task such as detectionor classification is as follows. The CNN in its first or bottomconvolution layer learns weights that allow the nodes to detect thepresence of local, primitive features such as lines, edges, and colorgradients. The output of that layer is a feature map that encodes (orrepresents) the input image with whatever features it learned duringtraining.

The role of each subsequent layer is to learn how to combine thefeatures presented from the previous layer to encode progressively moreglobal and abstract features into representations that will leadultimately to layers that combine features into objects of interest. Asimple example is a CNN that uses early layers to detect the presence oflines with various orientations. Middle (convolutional) layers learn todetect various combinations of these lines. Later (convolutional)layers, because of the down-sampling that happens with pooling, seelarger regions and can thus learn more abstract combinations: some nodeslearn to detect shapes that look like eyes, some nodes respond to shapesthat look like noses, etc. Finally, at the top of the network (near theoutput), the fully-connected (non-convolutional) layers learn to encodethe “rules” for how these objects need to be combined for the particulartask. If the task is to detect the presence of a face in the inputimage, the fully-connected layer learns how the preceding layer'srepresentation (eyes, nose, mouth, etc.) need to be configured toconstitute a well-formed face.

For the present problem domain in which we wish to detect features inmicroscopy images that are undetectable to humans, the task is quitedifferent than the typical kinds of tasks performed by CNNs such asdetection, classification, segmentation, etc. This task may be referredto as “supervised discovery.”

In addition to using previously known CNN structures and principles, thepresent disclosure presents new kinds of CNNs that may be better suitedfor the task of supervised discovery. In contrast to a detection orclassification task in which the objective is to progressively encodelow-level features in lower layers (closer to input) into more abstracthigh-level features in higher layers (closer to output), the objectivemay be to discover (and encode) very subtle, low-level pixel-spacefeatures that signify the presence of some condition that is present,but visually undetectable by a human.

To train a CNN to do supervised discovery, at least one embodimentpreserves all of the general CNN characteristics described above. Butthey may be implemented such that out-of-band (i.e., out-of-channel)ground truth mechanisms can be used to drive a learning process thatimproves mapping of low-level features to the out-of-band target signal(e.g., the features that aren't dependably distinguishable or detectableto human eyes). For example, rather than trying to abstract an image toget to a higher-level classification, the neural network may detect oneor more low level features that signify something about the patient(e.g., a diseased status).

In normal classification (e.g., based on human distinguishablefeatures), the lowest level convolutions are just the building blocksthat get abstracted out. With supervised discovery, we may not beinterested in building up a hierarchy of low-level features that formhigh-level features. Rather, we may wish to maintain veryhigh-resolution representations of the input image in the network'slayers and, as part of the training process, learn how these varioushigh-resolution encodings must be combined to learn the mapping to theout-of-band target signal. For “invisible features” a new layer, eitheras a part of the classification neural network or a follow on one may beused to once again looks at the image in full resolution with possiblynew convolutions. These new convolutions are used to find features thatsub-classify rather than being built upon to abstract higher levelfeatures. Using pre-classified images, either manually or from a neuralnetwork, as inputs into the layers to find invisible features willpossibly help the invisible layers find good features during trainingand improve accuracy during prediction. See, for example, FIG. 9.

Because the goal may be to find “invisible features” (e.g., notdetectable by even a trained human) and not abstract the image, lesspooling may be required in some embodiments. It is even possible thatexcessive pooling, such as that found in conventional CNNs for objectdetection/recognition, increases the risk of important feature loss thatreduces classification accuracy. A few pooling layers might still beused to help detect clustering or other patterns of lower levelfeatures, but this would be less than a normal classification model likeVGG16, at least in some embodiments. With a conventional CNN used fortypical kinds of tasks such as detection, classification, orsegmentation the pooling operation is used to down-sample therepresentation of the image as a way to encourage a kind of abstractingprocess from low-level to high-level features. But with superviseddiscovery the pooling operation takes on a more diminished role. Whilesome down-sampling may always be required to help form representationsfrom previous layers, supervised discovery embodiments disclosed hereinmay require that the CNN preserve a high-level representation even tothe higher layers. This means that some layers in a supervised discoveryCNN have the original or bottom layer as part of its input field.

While it may not always be feasible to determine exactly which featureswill be useful for classification, having a general idea influenceschoices made about neural network design. For example, at the conclusionof the training of a conventional CNN for tasks such as detection,classification or segmentation, it is common to attempt to identify theearly-layer features that were learned so as to help select parametersfor subsequent fine-tune training. However, with supervised discoverythe features, by definition, are low-level, pixel-space features thatfrequently would not lend themselves to simple summaries like “lineorientations”.

One example of an “invisible” feature set, consider the output of lightmicroscopy, in which there are features that are smaller than onemicron. Due to the diffraction limit of light, these features appearblurred. The information for that feature is still present, but it ispresent with a much lower signal-to-noise ratio. The conversion todigital bits as part of the digitization process further distorts thefeatures and also introduces noise that results in an even lowersignal-to-noise ratio. But if the learning algorithm is provided with asufficient number of training examples, with the right architecture, thenetwork can learn the significance of these features even if they appearto be invisible to a human. This is possible because, in many cases, CNNembodiments disclosed herein can separate signal from noise better thanhumans and can also find more complex patterns. Both cases may allowcomputers to classify beyond human ability.

Other examples of features of interest that may or may not be detectedby humans are minute changes in colors or color gradients, minuteschanges in the straightness or bumpiness or lines or curves, and/orvariation in speckle patterns a human might perceive as noise. For a CNNto detect these very fine-grain pixel-level features, the network mayrequire the following kinds of modifications not typically found in aconventional CNN: the use of bypass layers (to provide the input imageas part of the representation provided to upper layers of the network)(see FIG. 8, for example); the CNN may require a much higher number ofconvolution nodes (filters) in some layers; the CNN may require muchmore variation in the kernel sizes between layers (so as to provide theappropriate receptive field); and/or the CNN may use stackedconvolutional layers without the usual pooling layers (or with fewerpooling layers) so as to maintain the resolution to detect veryfine-grain patterns. One example embodiment of a specialized CNN ordetecting features which may not be noticed or detected by a human isillustrated in FIGS. 8-9.

FIG. 7 is a block diagram illustrating one embodiment of a CNN 700 forclassifying particles or particle types. The CNN 700 includes repeatingsets of convolutional layers 702, activation layers 704, and poolinglayers 706. The convolutional layers 702 and activation layers 704 maybe fully connected with the nodes of a previous layer. A classificationlayer 708 may compute one or more output classifications to be output bythe CNN 700. In one embodiment, the CNN 700 may be trained using imageswith ground truth derived from out-of-channel mechanisms. The CNN 700may be used for performing classifications that can be determined by ahuman. For example, the CNN 700 may be used for determining a cell typeclassification (e.g., as a white blood cell, red blood cell, etc.), orany detecting any other classifications.

FIG. 8 is a block diagram illustrating one embodiment of a CNN 800 forclassifying particles or particle types. The structure of the CNN 800may improve the ability of the CNN 800 (when compared to the CNN 700 ofFIG. 7) to detect features that are “invisible” to humans. The CNN 800includes repeating sets of convolutional layers 802 and activationlayers 804. The convolutional layers 802 may have increased kernel sizeand/or more filters than the convolutional layers of the CNN 700. Forexample, kernel size may have increased variation to provide anappropriate receptive field. After a few repetitions of convolutionallayers 802 and activation layers 804, a pooling layer 806 may be usedfor down sampling. Note that fewer pooling layers are used with relationto the number of convolutional layers 802 and activation layers 804.

The CNN 800 also includes connections feeding forward an output of anactivation layer to a non-subsequent later convolutional layer 802. Thismay bypass some of the convolutions and pooling to maintain originalpixel detail. Additional connections or bypass layers may be used tofeed forward an output from an initial activation layer 804, lateractivation layer 804, or original pixel values to later layers. In oneembodiment, the sequence illustrated in the CNN 800 may be repeated aplurality of times until a classification layer 808 computes and outputsa final classification.

FIG. 9 is a schematic block diagram illustrating classification of aspecimen in an image, according to one embodiment. An image of aspecimen may be input into an initial neural network classifier 902. Theimage may include a cropped microscopy image of a cell or other materialor particle. The initial neural network classifier 902 generates anintermediate classification. The intermediate classification and theoriginal image may be provided as input into a final neural networkclassifier 904. The final neural network classifier 904 generates andoutputs a final classification. In one embodiment, the initial neuralnetwork classifier 902 may have a structure similar to that depicted inrelation to the CNN 700 of FIG. 7 while the final neural networkclassifier 904 has a structure similar to the CNN 800 of FIG. 8.

The intermediate classification may be an initial classification that ismore general than a desired final classification. In one embodiment, theintermediate classification may generate a classification that can beverified by a human. For example, the intermediate classification maydistinguish between particle types or cell types that are alsodistinguishable by a human. By way of example, the intermediateclassification may be a classification of a cell type, e.g., a whiteblood cell versus a red blood cell. Other classifications may also bedetermined as part of the intermediate classification, withoutlimitation.

The final classification may be a better than human classification. Forexample, the final classification may distinguish betweenclassifications that a human could not do based on the image alone. Forexample, the final classification may be a classification of a whiteblood cell (as classified in the intermediate classification) as comingfrom a patient with a specific class of disease or from a patientwithout the specific class of disease. The final classification may bebased on features in the image that are not reliably detectable by ahuman but which the final neural network classifier has been trained todetect.

Referring now to FIG. 10, a block diagram of an example computing device1000 is illustrated. Computing device 1000 may be used to performvarious procedures, such as those discussed herein. Computing device1000 can function as a server, a client, classification system 104, anyother system or computing entity. Computing device 1000 can performvarious monitoring functions as discussed herein, and can execute one ormore application programs, such as the application programs describedherein. Computing device 1000 can be any of a wide variety of computingdevices, such as a desktop computer, a notebook computer, a servercomputer, a handheld computer, tablet computer and the like.

Computing device 1000 includes one or more processor(s) 1002, one ormore memory device(s) 1004, one or more interface(s) 1006, one or moremass storage device(s) 1008, one or more Input/Output (I/O) device(s)1010, and a display device 1030 all of which are coupled to a bus 1012.Processor(s) 1002 include one or more processors or controllers thatexecute instructions stored in memory device(s) 1004 and/or mass storagedevice(s) 1008. Processor(s) 1002 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 1004 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM) 1014) and/ornonvolatile memory (e.g., read-only memory (ROM) 1016). Memory device(s)1004 may also include rewritable ROM, such as Flash memory.

Mass storage device(s) 1008 include various computer readable media,such as magnetic tapes, magnetic disks, optical disks, solid-statememory (e.g., Flash memory), and so forth. As shown in FIG. 10, aparticular mass storage device is a hard disk drive 1024. Various drivesmay also be included in mass storage device(s) 1008 to enable readingfrom and/or writing to the various computer readable media. Mass storagedevice(s) 1008 include removable media 1026 and/or non-removable media.

I/O device(s) 1010 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 1000.Example I/O device(s) 1010 include cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, and the like.

Display device 1030 includes any type of device capable of displayinginformation to one or more users of computing device 1000. Examples ofdisplay device 1030 include a monitor, display terminal, videoprojection device, and the like.

Interface(s) 1006 include various interfaces that allow computing device1000 to interact with other systems, devices, or computing environments.Example interface(s) 1006 may include any number of different networkinterfaces 1020, such as interfaces to local area networks (LANs), widearea networks (WANs), wireless networks, and the Internet. Otherinterface(s) include user interface 1018 and peripheral device interface1022. The interface(s) 1006 may also include one or more user interfaceelements 1018. The interface(s) 1006 may also include one or moreperipheral interfaces such as interfaces for printers, pointing devices(mice, track pad, or any suitable user interface now known to those ofordinary skill in the field, or later discovered), keyboards, and thelike.

Bus 1012 allows processor(s) 1002, memory device(s) 1004, interface(s)1006, mass storage device(s) 1008, and I/O device(s) 1010 to communicatewith one another, as well as other devices or components coupled to bus1012. Bus 1012 represents one or more of several types of busstructures, such as a system bus, PCI bus, IEEE bus, USB bus, and soforth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 1000, and areexecuted by processor(s) 1002. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein.

The embodiments of systems, methods, and devices discussed herein may beapplied to a wide range of sample types for detection of variousparticles, materials, or the like. The following paragraphs describedifferent types of samples which may be imaged and identified usingmethods, systems, or devices disclosed herein.

Many of the examples and embodiments disclosed herein are directed tocytology or cytopathology. However, the principles disclosed herein mayalso be applied to numerous particle types and studies. The followingsections discuss other types of particles, materials, samples, studies,or classifications that can be used for machine learning training andclassification, according to numerous embodiments contemplated by thepresent disclosure.

Tissue

In one embodiment, training, classification, and identification ofparticles or materials within tissue may be performed. Tissue samplesmay be obtained from biopsies, tissue scraping, brushing, liquidremoval/withdrawal, or using any other tissue sampling or biopsymethods. Tissues samples may include human tissue samples or animaltissue samples. The tissue samples may be applied to a slide forimaging, or may be imaged in any other manner for a respective samplingor biopsy method.

Once images of the tissue sample are obtained, the images may beprovided to a classification system 104. The classification system 104may use one or more neural networks for classifying or detectingparticles or materials in the images. For example, the classificationsystem 104 may include neural networks that have been trained toidentify particles or materials that may be present in a tissue samplefrom a specific region or location of a body or organ. In oneembodiment, the neural networks may include neural networks configuredto identify cells, particles, or materials that indicate the presence ofa specific type of cancer. For example, a first neural network mayidentify cells corresponding to a specific type of cancer and a secondneural network may identify other molecules or cells corresponding tothe same or different type of cancer. The types of particles searchedfor, and thus, the neural networks used for classification, may dependon a specific type of examination. For example, the images may beprovided to the classification system 104 with an indication that anexamination is to be performed to detect a cancerous or benign nature ofthe tissue.

The classification system 104 may output an indication ofclassifications for particles within the images in the form of a heatmap, table, or the like. For example, the indication of classificationor identification may correspond to a specific region of an image,bounding box, or other location within the digitized images. Based onthe output, a report may be generated indicating the presence ofparticular particle types and their number, a diagnoses for a medicalcondition (e.g., such as cancer or other disease), recommended medicalprocedures or treatment, or the like. The report may be provided to amedical professional or patient for review.

Bone and Bone Marrow

In one embodiment, identification of particles or materials within bonesamples may be performed. Bone samples may include bone matter such asbone, bone marrow, or the like from humans or any other animal. Forexample, bone samples from pets, wildlife, farm animals, or otheranimals may be analyzed for health, bone density, or disease indicators.The bone samples may be applied to a slide for imaging, or may be imagedin any other manner for a respective sampling method. Once images of thebone sample are obtained, the images may be provided to a classificationsystem 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a bone sample from a human or other specific animal type. In oneembodiment, the neural networks may include neural networks configuredto identify blood cells such as red blood cells or white blood cells. Inone embodiment, the neural networks may include neural networksconfigured to classify the bone as having a specific bone density. Forexample, the neural networks may be trained to categorize a particle inthe bone sample as corresponding to a bone density above or below aspecific bone density threshold. In one embodiment, the bone samples maybe processed using a plurality of neural networks. For example, a firstneural network may identify the presence of red blood cells and/or whiteblood cells and a second neural network may determine a bone density forthe particles in an image. The types of particles searched for, and thusthe neural networks used for classification, may depend on a specifictype of examination. For example, the images may be provided to theclassification system 104 with an indication that an examination is tobe performed to detect particles or density of the bone sample thatindicate that the source patient or animal may have a specific medicalcondition or disease.

The classification system 104 may output an indication ofclassifications for particles within the images of the bone sample inthe form of a heat map, table, or the like. For example, the indicationof classification or identification may correspond to a specific regionof an image, bounding box, or other location within the digitizedimages. Based on the output, a report may be generated indicating thepresence of particular particle types and their count, a diagnosis for amedical condition, recommended medical procedures or treatment, or thelike. The report may be provided to a medical professional or patientfor review.

Fecal

In one embodiment, samples and identification of particles or materialswithin fecal matter are performed. Fecal samples may include fecalmatter from humans or any other animal. For example, fecal matter frompets, wildlife, farm animals, or other animals may be analyzed forhealth or disease indicators. The fecal samples may be applied to aslide for imaging, or may be imaged in any other manner for a respectivesampling method. Once images of the fecal sample are obtained, theimages may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a fecal sample from a human or other specific animal type. In oneembodiment, the neural networks may include neural networks configuredto identify one or more different types of parasites. In one embodiment,the neural networks may include neural networks configured to identifyblood or blood particulates. In one embodiment, the neural networks mayinclude neural networks configured to identify ovum (egg cells) in thefecal matter. For example, a first neural network may identify thepresence of blood, a second neural network may identify the presence ofovum, and a third neural network may identify one or more types ofparasites. The types of particles searched for, and thus, the neuralnetworks used for classification, may depend on a specific type ofexamination. For example, the images may be provided to theclassification system 104 with an indication that an examination is tobe performed to detect particles in the fecal sample that indicate thatthe source patient or animal may have a specific medical condition ordisease.

The classification system 104 may output an indication ofclassifications for particles within the images in the form of a heatmap, table, or the like. For example, the indication of classificationor identification may correspond to a specific region of an image,bounding box, or other location within the digitized images. Based onthe output, a report may be generated indicating the presence ofparticular particle types and their count, a diagnosis for a medicalcondition, recommended medical procedures or treatment, or the like. Thereport may be provided to a medical professional or patient for review.

Blood

In one embodiment, identification of particles or materials within bloodsamples may be performed. Blood samples may include blood samples drawnfrom humans or any other animal. For example, blood samples from pets,wildlife, farm animals, or other animals may be analyzed for health ordisease indicators. The blood samples may be applied to a slide forimaging, or may be imaged in any other manner for a respective samplingmethod. Once images of the blood sample are obtained, the images may beprovided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a blood sample from a human or specific animal type. In oneembodiment, the neural networks may include neural networks configuredto identify blood cells such as red blood cells or white blood cells. Inone embodiment, the neural networks may include neural networksconfigured to classify or detect blood cells as having abnormal shapes,such as sickle cells. For example, the neural networks may be trained tocategorize a particle in the blood sample as a sickle cell or normal redblood cell. In one embodiment, the blood samples may be processed usinga plurality of neural networks. For example, a first neural network mayidentify the presence of red blood cells, a second neural network mayidentify the presence of white blood cells, and a third neural networkmay identify or detect abnormally shaped cells, such as sickle cells.The types of particles searched for, and thus the neural networks usedfor classification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectblood cell count, blood cell abnormality, or the like.

The classification system 104 may output an indication ofclassifications for particles within the images of the blood sample inthe form of a heat map, table, or the like. For example, the indicationof classification or identification may correspond to a specific regionof an image, bounding box, or other location within the digitizedimages. Based on the output, a report may be generated indicating thepresence of particular particle types and their count (e.g., red orwhite blood cell count), a diagnosis for a medical condition,recommended medical procedures or treatment, or the like. The report maybe provided to a medical professional or patient for review.

Urine

In one embodiment, identification of particles or materials within urinesamples may be performed. Urine samples may include urine from humans orany other animal. For example, urine samples from pets, wildlife, farmanimals, or other animals may be analyzed for health, urine density, ordisease indicators. The urine samples may be applied to a slide forimaging, or may be imaged in any other manner for a respective samplingmethod. Once images of the urine sample are obtained, the images may beprovided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a urine sample from a human or other specific animal type. In oneembodiment, the neural networks may include neural networks configuredto identify blood particles or cells, such as red blood cells or whiteblood cells. In one embodiment, the neural networks may include neuralnetworks configured to identify sediment (such as protein, leukocytes,blood cells, or bacteria) in the urine. For example, sediment in urinemay indicate a higher likelihood of a urinary tract infection for apatient. In one embodiment, a neural network is configured to detect acrystal, mineral, kidney stone, or other object or particle in theurine. In one embodiment, the neural network for processing images ofurine samples is configured to detect any particulates and may classifyor detect the particulate types.

In one embodiment, the urine samples may be processed using a pluralityof neural networks. For example, a first neural network may identify thepresence of red blood cells and/or white blood cells and a second neuralnetwork may detect crystal, mineral, or kidney stone particles. Thetypes of particles searched for, and thus the neural networks used forclassification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectsymptoms of a urinary tract infection or kidney stones.

The classification system 104 may output an indication ofclassifications for particles within the images of the urine sample inthe form of a heat map, table, or the like. For example, the indicationof classification or identification may correspond to a specific regionof an image, bounding box, or other location within the digitizedimages. Based on the output, a report may be generated indicating thepresence of particular particle types and their count, a diagnosis for amedical condition, recommended medical procedures or treatment, or thelike. For example, the report may indicate the presence of particles orsediment that correlates with an infection or kidney stones. The reportmay be provided to a medical professional or patient for review.

Sputum

In one embodiment, identification of particles or materials withinsputum samples may be performed. Sputum samples may include saliva,mucus, or other material from an oral cavity or respiratory tract from ahuman or any other animal. For example, sputum samples from a humanpatient, pets, wildlife, farm animals, or other animals may be analyzedfor health or disease indicators. The sputum samples may be applied to aslide for imaging, or may be imaged in any other manner for a respectivesampling method. Once images of the sputum sample are obtained, theimages may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images of thesputum sample. For example, the classification system 104 may includeneural networks that have been trained to identify particles ormaterials that may be present in a sputum sample from a human or otherspecific animal type. In one embodiment, the neural networks may includeneural networks configured to identify tuberculosis (TB), streptococcalpharyngitis (i.e. strep), saliva, nasal mucus, or other particles ormaterials present in a sputum sample. For example, the neural networksmay be configured to identify cells, bacteria, or other particles ormaterials that may be indicators of a disease or other medical conditionof a patient. In one embodiment, the sputum samples may be processedusing a plurality of neural networks that have been specifically trainedor are being trained to detect a specific type of cell, bacteria,particle, or material. The types of particles searched for, and thus theneural networks used for classification, may depend on a specific typeof examination. For example, the images may be provided to theclassification system 104 with an indication that an examination is tobe performed to detect evidence of a specific disease, mucosis, or othercondition.

The classification system 104 may output an indication ofclassifications for particles within the images of the sputum sample inthe form of a heat map, table, or the like. For example, the indicationof classification or identification may correspond to a specific regionof an image, bounding box, or other location within the digitizedimages. Based on the output, a report may be generated indicating thepresence of particular particle types and their count, a diagnosis for amedical condition, recommended medical procedures or treatment, or thelike. The report may be provided to a medical professional or patientfor review.

Infectious Disease

In one embodiment, identification of particles or materials within aninfectious disease sample may be performed. Infectious disease samplesmay include liquid, tissue, blood, waste, or any other material from thebody of a human, animal, or plant. For example, infectious diseasesamples may be analyzed for health, infectious disease density, ordisease indicators. The infectious disease samples may be applied to aslide for imaging, or may be imaged in any other manner for a respectivesampling method. Once images of the infectious disease sample areobtained, the images may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin an infectious disease sample. In one embodiment, the neural networksmay include neural networks configured to identify particles, cells orbacteria indicating the presence of one or more of malaria,tuberculosis, bone tuberculosis, red blood cell abnormalities,gonorrhea, chlamydia, yeast, trichomoniasis, babesia, bartonella,Howell-Jolly bodies, Papenheimer bodies, viral bodies, or other bacteriaor particles. In one embodiment, the infectious disease samples may beprocessed using a plurality of neural networks that each search for adifferent type or classification of bacteria or particle. The types ofparticles searched for, and thus the neural networks used forclassification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectparticles that indicate that the source patient, animal, or plant mayhave a specific medical condition or disease.

The classification system 104 may output an indication ofclassifications for particles within the images of the infectiousdisease sample in the form of a heat map, table, or the like. Forexample, the indication of classification or identification maycorrespond to a specific region of an image, bounding box, or otherlocation within the digitized images. Based on the output, a report maybe generated indicating the presence of particular particle types andtheir count, a diagnosis for a medical condition or infectious disease,recommended medical procedures or treatment, or the like. The report maybe provided to a medical professional or patient for review.

Cervical Mucus

In one embodiment, identification of particles or materials withincervical mucus samples may be performed. Cervical mucus samples mayinclude cervical mucus samples obtaining using a pap smear from humansor any other animal. The cervical mucus samples may be applied to aslide for imaging, or may be imaged in any other manner for a respectivesampling method. Once images of the cervical mucus sample are obtained,the images may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a cervical mucus sample from a human or other specific animal type.For example, the neural networks may identify cells that displaypotentially pre-cancerous changes, such as cervical intraepithelialneoplasia (CIN) or cervical dysplasia, the squamous intraepitheliallesion system (SIL), or the like.

In one embodiment, the neural networks may include neural networksconfigured to identify cancerous cell or particles or cells indicatingthe presence of cancer. In one embodiment, the cervical mucus samplesmay be processed using a plurality of neural networks to check for oneor more different types of cancerous cells.

The classification system 104 may output an indication ofclassifications for particles within the images of the cervical mucussample in the form of a heat map, table, or the like. For example, theindication of classification or identification may correspond to aspecific region of an image, bounding box, or other location within thedigitized images. Based on the output, a report may be generatedindicating the presence of particular particle types and their count, adiagnosis for a medical condition (e.g., such as the presence ofcancer), recommended medical procedures or treatment, or the like. Thereport may be provided to a medical professional or patient for review.

Vaginal Fluid

In one embodiment, identification of particles or materials withinvaginal fluid samples may be performed. Vaginal fluid samples mayinclude fluid from vaginal discharge of a human or any other animal. Thevaginal fluid samples may be applied to a slide for imaging, or may beimaged in any other manner for a respective sampling method. Once imagesof the vaginal fluid sample are obtained, the images may be provided toa classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a vaginal fluid sample from a human or other specific animal type. Inone embodiment, the neural networks may include neural networksconfigured to identify cells, bacteria, spores or other material orparticles indicating the presence of bacterial vaginosis, candida,and/or gardnerella. In one embodiment, the vaginal fluid samples may beprocessed using a plurality of neural networks. For example, a firstneural network may identify the presence of bacterial vaginosis, asecond neural network may identify the presence of candida, and a thirdneural network may identify the presence of gardenalia. The types ofparticles searched for, and thus the neural networks used forclassification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectparticles or density of the vaginal fluid sample that indicate that thesource patient or animal may have a specific medical condition ordisease.

The classification system 104 may output an indication ofclassifications for particles within the images of the vaginal fluidsample in the form of a heat map, table, or the like. For example, theindication of classification or identification may correspond to aspecific region of an image, bounding box, or other location within thedigitized images. Based on the output, a report may be generatedindicating the presence of particular particle types and their count, adiagnosis for a medical condition, recommended medical procedures ortreatment, or the like. The report may be provided to a medicalprofessional or patient for review.

Milk

In one embodiment, identification of particles or materials within milksamples may be performed. Milk samples may include breast milk liquid orsamples from humans or any other animal. The milk samples may be appliedto a slide for imaging, or may be imaged in any other manner for arespective sampling method. Once images of the milk sample are obtained,the images may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a milk sample from a human or other specific animal type. In oneembodiment, the neural networks may include neural networks configuredto identify parasites, somatic cells (e.g., white blood cells), or thelike. In one embodiment, the milk samples may be processed using aplurality of neural networks. For example, a first neural network mayidentify the presence of a first type of parasite, a send neural networkmay identify the presence of a second type of parasite, and at thirdneural network may identify the presence of white blood cells. The typesof particles searched for, and thus the neural networks used forclassification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectparticles or density of the milk sample that indicate that the sourcepatient or animal may have a specific medical condition or disease.

The classification system 104 may output an indication ofclassifications for particles within the images of the milk sample inthe form of a heat map, table, or the like. For example, the indicationof classification or identification may correspond to a specific regionof an image, bounding box, or other location within the digitizedimages. Based on the output, a report may be generated indicating thepresence of particular particle types and their count, a diagnosis for amedical condition, recommended medical procedures or treatment, or thelike. The report may be provided to a medical professional or patientfor review.

Semen

In one embodiment, identification of particles or materials within semensamples may be performed. Semen samples may include semen matter fromhumans or any other animal. For example, semen samples from pets,wildlife, farm animals, or other animals may be analyzed for health,semen count, or disease indicators. The semen samples may be applied toa slide for imaging, or may be imaged in any other manner for arespective sampling method. Once images of the semen sample areobtained, the images may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a semen sample from a human or other specific animal type. In oneembodiment, the neural networks may include neural networks configuredto identify sperm morphology or other indications of fertility of thesource patient or animal. In one embodiment, the semen samples may beprocessed using a plurality of neural networks. For example, a firstneural network may identify the presence of normal sperm and a secondneural network may determine a presence of abnormal sperm. The types ofparticles searched for, and thus the neural networks used forclassification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectparticles or sperm count that may indicate that the source patient oranimal may have a specific medical condition or disease.

The classification system 104 may output an indication ofclassifications for particles within the images of the semen sample inthe form of a heat map, table, or the like. For example, the indicationof classification or identification may correspond to a specific regionof an image, bounding box, or other location within the digitizedimages. Based on the output, a report may be generated indicating thepresence of particular particle types and their count (e.g., sperm countor percentage of abnormal sperm), a diagnosis for a medical condition,recommended medical procedures or treatment, or the like. The report maybe provided to a medical professional or patient for review.

Geology

In one embodiment, identification of particles or materials withingeological samples may be performed. Geological samples may includedirt, soil, rock, oil or other liquid or other material from the earthor other geological or ecological location. The geological samples maybe applied to a slide for imaging, or may be imaged in any other mannercorresponding to a respective sampling method. Once images of thegeological sample are obtained, the images may be provided to aclassification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a geological sample. In one embodiment, the neural networks mayinclude neural networks configured to identify dinoflagellates, specificmineral or crystal types, oil, types of soil particles (sand, clay, orbiological matter), or the like. In one embodiment, the geologicalsamples may be processed using a plurality of neural networks. Forexample, a first neural network may identify the presence of oilparticles or molecules and a second neural network may determine thepresence of specific types of minerals or crystals. The types ofparticles searched for, and thus the neural networks used forclassification, may depend on a specific type of examination.

The classification system 104 may output an indication ofclassifications for particles within the images of the geological samplein the form of a heat map, table, or the like. For example, theindication of classification or identification may correspond to aspecific region of an image, bounding box, or other location within thedigitized images. Based on the output, a report may be generatedindicating the presence of particular particle types and their count, asoil health condition or a specific purpose (e.g., crops), a recommendedremediation procedure, or the like. The report may be provided to afarmer or scientist for review.

Forensics

In one embodiment, identification of particles or materials withinforensic samples may be performed. Forensic samples may include forensicmatter from a crime scene or other forensic scene or investigation. Forexample, forensic samples from carpet, furniture, floors, clothes,shoes, tires, jewelry, hair, mail, or the like may be obtained forlegal, historical, or other purposes. The forensic samples may beapplied to a slide for imaging, or may be imaged in any other manner fora respective sampling method. In one embodiment, camera images using aconventional camera may be obtained. For example, camera images of tiretracks, fingerprints, shoe footprints, tracks, or other evidence may betaken and analyzed. Once images of the forensic sample are obtained, theimages may be provided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin a forensic sample. In one embodiment, the neural networks may includeneural networks configured to identify drugs, anthrax, blood, minerals,fibers, clothing, shoe rubber, crystals, semen, hair, plant matter,biological mater, or other particles or substances. In one embodiment,the neural networks may identify a type of particle or material. In oneembodiment, neural networks or matching algorithms may be used to matchdrugs, anthrax, blood, minerals, fibers, clothing, shoe rubber,crystals, semen, hair, plant matter, biological mater, or otherparticles or substances at one location with samples or materialobtained from another location. For example, shoeprints, shoe patterns,fingerprints, tires (tire tracks), fibers, clothing, hair, semen, blood,or other matter may be matched with those of a subject or suspect. Inone embodiment, an image of a forensic sample and a reference image(e.g., of known material gathered from another location or suspect) maybe fed into a neural network for detecting the similarity or matching ofthe particles, materials, or items.

The types of particles searched for, and thus the neural networks usedfor classification, may depend on a specific type of examination. Forexample, the images may be provided to the classification system 104with an indication that an examination is to be performed to detectparticles or density of the forensic sample that indicate that thesource patient or animal may have a specific medical condition ordisease.

The classification system 104 may output an indication ofclassifications for particles within the images of the forensic samplein the form of a heat map, table, or the like. For example, theindication of classification or identification may correspond to aspecific region of an image, bounding box, or other location within thedigitized images. Based on the output, a report may be generatedindicating the presence of particular particle types and their count, asimilarity between particles, materials, or items, or the like. Thereport may be provided to an investigator or customer for review.

Agriculture

In one embodiment, identification of particles or materials withinagriculture samples may be performed. Agriculture samples may includeagriculture matter from soils (e.g., soil samples), dirt, fertilizers,or other materials used during the planting and growing of crops. Theagriculture samples may be applied to a slide for imaging, or may beimaged in any other manner for a respective sampling method. Once imagesof the agriculture sample are obtained, the images may be provided to aclassification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify particles or materials that may be presentin an agriculture sample. In one embodiment, the neural networks mayinclude neural networks configured to particles (or organisms) such asnematodes, pollen, bacteria, protozoa, cancerous cells, or the like in asoil sample or other agriculture sample. In one embodiment, theagriculture samples may be processed using a plurality of neuralnetworks. For example, a first neural network may identify the presenceof nematodes, a second neural network may identify the presence ofpollen, and a third neural network may identify the presence ofbacteria, or the like. The types of particles searched for, and thus theneural networks used for classification, may depend on a specific typeof examination.

The classification system 104 may output an indication ofclassifications for particles within the images of the agriculturesample in the form of a heat map, table, or the like. For example, theindication of classification or identification may correspond to aspecific region of an image, bounding box, or other location within thedigitized images. Based on the output, a report may be generatedindicating the presence of particular particle types and their count, adiagnosis for soil or plant health, a recommended remediation procedureor soil treatment, or the like.

Insect or Small Animal

In one embodiment, identification of particles or materials withininsect or small animal samples may be performed. Insect or small animalsamples may include full bodies, portions of bodies, larvae, eggs, orthe like of insects or small animals found in soil, air, water, food,plant, ecological environment, or other location. For example, thesamples may include adult, egg, larvae, or other life stages of insectsor small animals. The insect or small animal samples may be applied to aslide for imaging, or may be imaged in any other manner for a respectivesampling method. For example, the insect/animal bodies, eggs, larvae orthe like may be (knowingly or unknowingly) gathered separately or withinanother material, such as a dirt, liquid, or other material. Once imagesof the insect or small animal sample are obtained, the images may beprovided to a classification system 104.

The classification system 104 may use one or more neural networks forclassifying or detecting particles or materials in the images. Forexample, the classification system 104 may include neural networks thathave been trained to identify the body, the portions of a body of aninsect or small animal, or the eggs or larvae within the insect or smallanimal sample. In one embodiment, the neural networks may include neuralnetworks configured to identify adult insects, eggs, larvae. Forexample, neural networks may be used to identify an adult mosquito, amosquito egg or larvae, or the like. In one embodiment, a neural networkmay distinguish between a male and female mosquito (or other insect oranimal). In one embodiment, the neural networks may include neuralnetworks configured to identify adult insects (or their larvae or eggs)of different types. For example, mosquitoes, bed bugs, tics, or otherinsects may be identified and classified.

In one embodiment, the neural networks may include neural networksconfigured to identify and classify or detect small animals, such asnematodes, and/or their eggs. In one embodiment, the neural networks mayinclude neural networks configured to classify or detect the insect orsmall animal as of a specific type so that different types ofundesirable insects or animals can be distinguished from benign orbeneficial insects or animals.

In one embodiment, the insect or small animal samples may be processedusing a plurality of neural networks. For example, a first neuralnetwork may identify the presence of a specific type of mosquito orinsect, second neural network identifies the presence of a specific typemosquito or insect larva or egg, and a third neural network may identifythe presence of a specific type of animal. The types of particlessearched for, and thus the neural networks used for classification, maydepend on a specific type of examination. For example, the images may beprovided to the classification system 104 with an indication that anexamination is to be performed to detect mosquito eggs or larvae.

The classification system 104 may output an indication ofclassifications for particles within the images of the insect or smallanimal sample in the form of a heat map, table, or the like. Forexample, the indication of classification or identification maycorrespond to a specific region of an image, bounding box, or otherlocation within the digitized images. Based on the output, a report maybe generated indicating the presence of particular particle types andtheir count (e.g., mosquito larvae count), recommended remediationprocedures or treatment for a location, or the like.

Various techniques, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, a non-transitorycomputer readable storage medium, or any other machine readable storagemedium wherein, when the program code is loaded into and executed by amachine, such as a computer, the machine becomes an apparatus forpracticing the various techniques. In the case of program code executionon programmable computers, the computing device may include a processor,a storage medium readable by the processor (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. The volatile and non-volatile memoryand/or storage elements may be a RAM, an EPROM, a flash drive, anoptical drive, a magnetic hard drive, or another medium for storingelectronic data. One or more programs that may implement or utilize thevarious techniques described herein may use an application programminginterface (API), reusable controls, and the like. Such programs may beimplemented in a high-level procedural or an object-oriented programminglanguage to communicate with a computer system. However, the program(s)may be implemented in assembly or machine language, if desired. In anycase, the language may be a compiled or interpreted language, andcombined with hardware implementations.

It should be understood that many of the functional units described inthis specification may be implemented as one or more components, whichis a term used to more particularly emphasize their implementationindependence. For example, a component may be implemented as a hardwarecircuit comprising custom very large scale integration (VLSI) circuitsor gate arrays, off-the-shelf semiconductors such as logic chips,transistors, or other discrete components. A component may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices, orthe like.

Components may also be implemented in software for execution by varioustypes of processors. An identified component of executable code may, forinstance, include one or more physical or logical blocks of computerinstructions, which may, for instance, be organized as an object, aprocedure, or a function. Nevertheless, the executables of an identifiedcomponent need not be physically located together, but may includedisparate instructions stored in different locations that, when joinedlogically together, include the component and achieve the stated purposefor the component.

Indeed, a component of executable code may be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within components, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. The components may be passive or active, including agentsoperable to perform desired functions.

Implementations of the disclosure can also be used in cloud computingenvironments. In this application, “cloud computing” is defined as amodel for enabling ubiquitous, convenient, on-demand network access to ashared pool of configurable computing resources (e.g., networks,servers, storage, applications, and services) that can be rapidlyprovisioned via virtualization and released with minimal managementeffort or service provider interaction, and then scaled accordingly. Acloud model can be composed of various characteristics (e.g., on-demandself-service, broad network access, resource pooling, rapid elasticity,measured service, or any suitable characteristic now known to those ofordinary skill in the field, or later discovered), service models (e.g.,Software as a Service (SaaS), Platform as a Service (PaaS),Infrastructure as a Service (IaaS)), and deployment models (e.g.,private cloud, community cloud, public cloud, hybrid cloud, or anysuitable service type model now known to those of ordinary skill in thefield, or later discovered). Databases and servers described withrespect to the disclosure can be included in a cloud model.

Reference throughout this specification to “an example” means that aparticular feature, structure, or characteristic described in connectionwith the example is included in at least one embodiment of the presentdisclosure. Thus, appearances of the phrase “in an example” in variousplaces throughout this specification are not necessarily all referringto the same embodiment.

As used herein, a plurality of items, structural elements, compositionalelements, and/or materials may be presented in a common list forconvenience. However, these lists should be construed as though eachmember of the list is individually identified as a separate and uniquemember. Thus, no individual member of such list should be construed as ade facto equivalent of any other member of the same list solely based onits presentation in a common group without indications to the contrary.In addition, various embodiments and examples of the present disclosuremay be referred to herein along with alternatives for the variouscomponents thereof. It is understood that such embodiments, examples,and alternatives are not to be construed as de facto equivalents of oneanother, but are to be considered as separate and autonomousrepresentations of the present disclosure.

Although the foregoing has been described in some detail for purposes ofclarity, it will be apparent that certain changes and modifications maybe made without departing from the principles thereof. It should benoted that there are many alternative ways of implementing both theprocesses and apparatuses described herein. Accordingly, the presentembodiments are to be considered illustrative and not restrictive.

Those having skill in the art will appreciate that many changes may bemade to the details of the above-described embodiments without departingfrom the underlying principles of the disclosure. The scope of thepresent disclosure should, therefore, be determined only by the claims,if any.

What is claimed is:
 1. A method of classifying microscopy imagescomprising: inputting an original microscopy image of a specimen into aninitial neural network classifier of a convolutional neural network;wherein the initial neural network classifier comprises one or morerepeating sets of an initial convolutional layer, an initial activationlayers, and an initial pooling layer, and wherein the originalmicroscopy image comprises one or more invisible features; wherein theinitial neural network classifier of the convolutional neural network istrained using images with ground truth derived from out-of-channelmechanisms; wherein the convolutional neural network includes one ormore processing devices which detect the one or more invisible featureswithin the original microscopy image, identify the one or more invisiblefeatures within the original microscopy image, and provide theidentified one or more of the invisible features to a classificationsystem to identify common invisible features between the originalmicroscopy image and a plurality of microscopy images, theclassification system: generating an intermediate classification fromthe original microscopy image using the initial neural networkclassifier; inputting the intermediate classification and the originalmicroscopy image into a final neural network classifier; wherein thefinal neural network classifier comprises one or more finalconvolutional layers, one or more final activation layers, and one ormore final pooling layers; wherein the final neural network classifiercomprises one or more sets of any combination of one or more finalconvolutional layers, one or more final activation layers, and one ormore final pooling layers; wherein the final neural network classifiercomprises one or more bypass layers to feed forward an initial, finalclassification from the final activation layer to the finalconvolutional layer thereby bypassing the pooling layer; and wherein thefinal convolutional layer of the final neural network classifier has anincreased kernel size and more filters than the initial convolutionallayer of the initial neural network classifier; generating one or morefinal classifications from the intermediate classification and theoriginal microscopy image, wherein the one or more final classificationsis based on the invisible features in the original microscopy image thatare not reliably detectable by a human that the final neural networkclassifier has been trained to detect; and outputting the one or morefinal classifications.